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Abstract 

Presently, many people are listening to music through 
streaming services. Though listening to music in such a way is 
convenient, we feel it lacks the anticipation of encountering 
unknown music like we felt by looking at the aloum covers. 
In addition, listening to a vinyl record requires some errands; 
however, the effort yields a rich and immersive musical ex- 
perience. In this paper, we design an alternative way to ap- 
preciate music that would provide a similar anticipation and 
immersive experience to encounter music. In the proposed 
experience, users perceive multiple 3D objects as crystal- 
lized music and select a song from them. When an object is 
submerged in water, bubbles are generated as part of the ob- 
ject dissolves, and music begins to flow. We developed an in- 
teractive system that realizes such an experience by creating 
objects, each with an IC tag inside, and preparing a water tank 
with a tag reader on the bottom to identify the object. Mu- 
sic objects are created from the sound source and molded 
by using 3D printer. We tested the interaction and confirmed 
that the system functions well as expected. We also ran a 
questionnaire to acquire users’ feedback for designing a bet- 
ter experience. Our objective with this research is to design 
a novel way to appreciate music. The experience consists of 
listening to music and appreciating it also by sight, by mate- 
rializing the tune as a 3D object. Digital technology has gen- 
erally been used to achieve fast and convenient functions. In 
contrast, this work attempts to design a “slow” experience in 
which one is confronted with music more carefully. 


Author keywords 
Digital Content; Interaction with Music; Interaction Design; 
UX Design 


Introduction 

We have encountered musical compositions in many differ- 
ent ways. When listening to vinyl records, we have sometimes 
chosen a song based on a preference for its cover. Presently, 
people encounter new music through a streaming service by 
selecting a tune directly or following suggestions and play- 
lists created by someone else. The latter seems to reduce 


the number of unexpected encounters with unknown music. 
We think there was the pleasure of collecting music before, 
by possessing the records and discs. However, such pleasure 
also seemed lost as people began to listen to music through 
streaming. Observing such situations, we devised an idea to 
conceptualize music as a tangible entity when we generally 
regard them as digital data now. We decided to design music 
as a crystalized object that produces sound when dipped in 
water and dissolving. We expect users to feel the music melt- 
ing out of the object. We aim here to design an alternative ex- 
perience of appreciating music and develop a system to re- 
alize it. We produce an object of music tune by analyzing the 
sound source and converting the music to a 3D shape as per 
a specific rule. The crystal’s appearance helps in imagining 
music, as the shape represents the characteristic of music. In 
addition, we think one could feel like possessing music more 
directly than owning it as a digital file. We expect this system 
to offer a novel experience for encountering music, not only 
providing entertainment. This research can also be regarded 
as an attempt to design a new experience by giving different 
attributes from its original perception, allowing the viewer to 
appreciate the subject from a different perspective. 

To realize the idea, we develop a system that identifies mu- 
sic tunes by objects. The system equips a water bowl into which 
a user throws the object. The object is molded by 3D printer, and 
an IC tag is installed inside it. The tag is read by a tag reader at 
the bottom of the bowl, which lets the system match the object 
with specific music. To make a representation of music melt- 
ing out, we insert a material like dry ice into the object and let it 
produce bubbles when submerged in water. We examined the 
interaction with the developed system and multiple objects, 
each materializing different music, and confirmed that specif- 
ic music is played by the corresponding object. To evaluate the 
experience provided by this study, we conducted a survey and 
administered a questionnaire to examine whether the system 
offered an engaging approach to music. 


Experience with Crystalized Music 
We considered what if we could crystalize a piece of music 
so one can take it by hand. Since the crystallized object is 


music, we should be able to expect to listen to the tune by 
dissolving it. As it would be natural to expect a different shape 
from different music (Figure 1), you will have fun imagining 
the original music from its shape. To make it happen, we de- 
signed a mechanism for the interaction. We also prepared a 
method to transform a music tune into 3D geometry for cre- 
ating crystalized music objects. 


Figure 1. Concept of crystalized music. 


We expect a user would have the experience shown in Fig- 
ure 2. First, the experience begins with viewing multiple mu- 
sic objects. Each object represents a different music. As the 
shapes represent certain musical features, they are generally 
different in their looks; some are spiky, whereas others are 
smooth. Therefore, the user can imagine the tune from the 
object's shape. A user would choose one to listen, not know- 
ing what music it is. Once chosen, one can throw the object 
into the water to dissolve it. The user will see the bubbles 
forming as the object dissolves, and the music starts to play 
with a lighting effect appearing simultaneously. When it is re- 
solved entirely, the sound will cease. 


We aim with this research to propose a different perspective 
of being involved with music. First, we want to provide a sen- 
sation of feeling music crystalized as an object. This expres- 
sion realizes the transformation of music, a temporary phe- 
nomenon, into something static and tangible that one can 
appreciate before listening to the sound. In addition, users 
can feel the music by watching and touching the object. That 
would allow users to select a tune to listen to in a different 
approach than generally practiced. We expect this interac- 
tion would lead to emphasizing the music's transient nature 
because users will observe the event also by sight. 


To establish an immersive experience with our approach, 
we wish to have the shapes somewhat reflect the music's 
characteristics. However, we do not want to create an object 
referring to a personal impression that would differ for each 
individual. Therefore, we need a method that automatically 
transforms the music into the sculpture design according to 
a particular rule or algorithm. We started by employing a sim- 


Figure 2. Entire process of the experience. 
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ple transformation rule that converts the sound wave pat- 
tern to a 3D object. By finding an appropriate transformation 
rule, people could envisage music by browsing the objects’ 
shapes. One could find similar music by sight but not through 
recommendations provided by a streaming service. That 
would be another different experience this approach would 
provide. Seeking an ideal algorithm is a topic for future work. 


Related Research and Works 

Various ways of interacting with music have been proposed. 
Some make the experience an installation that facilitates a 
specific space to appreciate listening to and interacting with 
the music. The Intelligent Street is an installation work that 
designs a space for interactively listening to music, where 
people can affect the music by sending text commands via 
smartphone (Lörstad, 2004). Sound Forest is another instal- 
lation work, a room with numerous illuminating interactive 
strings (Frid, 2019). Multiple people can play with a string si- 
multaneously to emit a sound and share the experience. May- 
er et al. (2014) have developed a system in which two users 
collaborate to create music and visual effects by interacting 
with two-sized semi-sphere domes. While these works and 
ours share the objective of attempting to design a novel expe- 
rience of interacting with music, theirs also aim to provide in- 
teraction among the individuals entertaining the experience. 


From an interaction design perspective, our approach can be 
considered a variation of designing a tangible user interface 
(TUI). A first of that kind happens to take a theme related to 
music (Ishii, 2004). Our work shares the concept of provid- 
ing an alternative experience relating to music. However, with 
Ishii’s Music Bottle, music is still the same sound phenome- 
non, though the interaction with music differs from regular 
listening practices. Our approach is different in making music 
a tangible entity to interact with. reacTable (Kaltenbrunner, 
2006) is a synthesizer system employing a tangible user in- 
terface (TUI) to create sound. The user synthesizes a sound 
by putting circular and cubic objects on a circular table. The 
visual effects of the objects appear on the table when they 
are placed on the table. When positioning multiple objects in 
a neighborhood, their effects link in creating a sound. There is 
still other research on designing specific tangible interfaces. 
“The Table is The Score” (Levin, 2006) proposes to use spec- 
trogram shapes formed by a rubber or felt and place them 
on the table to play sound. In adopting TUI, many approach- 
es seem to treat objects as interfaces. In the case of Ishii’s 
Music Bottle, the interaction is designed by fitting music into 
the metaphor of a bottle, as a thing to keep something in it. 
We believe that the interactions with a TUI must be designed 
so that the object is well integrated with the context of the 
content, rather than simply being used as an interface. We 
conducted this study considering this concept. 


In realizing this work, we have sculpted the music as a three-di- 
mensional object. Similarly, several attempts have been made 
to transform music into a visual representation. Narratives 2.0 
(Dittrich, 2008) is an example of music visualization. It uses 
the frequency of a music channel to determine the turns of 
lines and colors in creating a visual representation. Another 
visualization work created for a poster (Bremer, 2020) utiliz- 
es the music's data and includes information on how the tune 
has been listened to at a streaming service. In addition, var- 


ious methods have been proposed for converting music into 
visual representation (Lima, 2021, Khulusi, 2020). Macrogroove 
(Chable, 2019) attempts to interactively convert sound into 3D 
shapes. A common approach among these studies and ours is 
that the shapes and colors are automatically generated from 
the music data according to a specific rule rather than through 
individual tastes. The resulting expression may not conform 
to human senses; however, it may lead to discoveries about a 
tune, providing a different perspective on appreciating music. 


Implementation 


System for Playing Sound 

The system equips a water bowl where users can drop the 
object. To realize the experience explained in the previous 
section, we need to implement the following functions for 
building the system. 


Identification of objects 

Detection of an object submerged in water 
Control of playing sound 

Illuminating effect (optional) 


ADE 


First, the system should be able to identify the object chosen 
for playing music. Several approaches can be used to identify 
the object. In this study, we inserted an IC tag into the objects. 
We employed this method because the system can simulta- 
neously detect an object's immersion in the water by placing 
a tag reader at the bottom of the water bowl. 


@ IC-tag reader 
@ Speaker 
® Projector 
(4) Camera 
(5) PC 

(6) Water tank 


Figure 3. System overview 


When the system detects an object submerged in water 
and identifies the corresponding music tunes, the system 
starts playing the sound via a speaker at the bottom of the 
system. Simultaneously, the system projects a visual effect 
onto the surrounding plane as a supplemental representation 
of sound. An extra function is required to stop the music by 
observing the melting progress. However, we have not imple- 
mented it yet. It is a topic for future work. Figure 3 illustrates 
an overview of the system. 


Crafting of Music Objects 
We need the followings for preparing the music object. 
1. Conversion of sound data to a 3D shape 
2. A mechanism to identify an object 
3. A mechanism to make the object look soluble in water 
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We do not want to manually design a 3D shape of each mu- 
sic by relying on individual sensitivity and preference. It would 
be ideal to have an automatic translating rule to create the 
objects of many different musical tunes. We are currently 
adopting a method that transforms the sound volume and 
pitch changes into a 3D object. The objects are then fabricat- 
ed using a 3D printer (Figure 4). We make a hole in the ob- 
ject for placing an IC tag, as shown in Figure 5. We also place 
a soluble material in that space. This mechanism generates 
bubbles, creating the appearance that the object is dissolving 
in water. The object itself remains unaltered even after the 
bubbles subside with the current setup, though. We also at- 
tempt to prepare molds and create objects out of meltable 
materials. However, placing an IC tag inside an object created 
this way is awkward because the tag remains after it melts. 
Therefore, we need to design another system for object iden- 
tification, which we have not implemented yet. 


Figure 4. Conversion of music data to a 3D shape. 


space to hold 
the material 
A and tag 


soluble material 


IC-tag 


Figure 5. Structure of the music object 


Evaluation and Discussion 

To verify that the system functions properly, we threw the 
objects into the water to examine if it correctly identified 
them. We threw three different objects into the tank in ran- 
dom order 20 times, with 16 times success. The failures 
were not due to misrecognition but because the function to 
recognize them was not activated properly. We interviewed 
11 persons on their experience with the system. 10 answered 
that it was an entertaining and impressive experience, and 
eight answered that they felt music was melted out from the 
object. The results suggested that the intention of the expe- 
rience was conveyed to the participants, though we consider 
further improvements are necessary to provide the experi- 
ence more realistic. 


This work offers the users largely two different experienc- 
es. One is choosing an object by imagining the music by its 
shape, as shown on the left image of Figure 6. The other is to 
feel the music as if it is melting from a solid object (Figure 
6, right, Figure 7). To make the first experience more afflu- 
ent, we believe the object's shape should reflect the music’s 
characteristics more. It would be possible to extract vari- 
ous other attributes that give the music its characteristics, 
such as the length of notes and the frequency of specific 
scale changes on the score. In this study, we only considered 
shapes to represent a piece of music; however, attributes 
like color, tactility, and weight could also be regarded as rep- 
resentational elements. Reflecting these in the model is an 
issue for future research. 


We have added some small interaction designs to reinforce 
the feeling of listening to crystallized music. After the system 
detects an object is submerged, an interval is inserted before 
the music begins, making the sound start at a low volume and 
increase in volume over time to match the amount of bub- 
bles. Another design is applied to the music's sound. If the 
sound is clearly audible, users cannot feel the music is play- 
ing underwater. Therefore, in addition to placing the speaker 
at the bottom of the water tank, we have digitally converted 
the music so that it sounds like it is playing underwater. These 
small designs are not necessary to achieve functionality. 
However, when creating an experience, we believe it essential 
to mind these small details. Currently, the object remains af- 
ter the bubble and music ceased because it is plastic. Howev- 
er, we want the object to disappear to express the one-time 
nature of crystallized music. We consider this would alter the 
experience further. 


Conclusion 

To design a new experience of appreciating music, we de- 
vised the concept of listening to music by melting a crystal- 
ized object. The innovation of this study is designing music 
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Figure 7. Bubbles coming out of the object 
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